

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>crimson &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/doctools.js"></script>
        <script src="../../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="osd" href="../osd/" />
    <link rel="prev" title="Crimson developer documentation" href="../" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../../internals/">Ceph 内幕</a></li>
          <li class="breadcrumb-item"><a href="../">Crimson developer documentation</a></li>
      <li class="breadcrumb-item active">crimson</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../../_sources/dev/crimson/crimson.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../balancer-design/">Ceph 如何均衡（读写、容量）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../crush-msr/">CRUSH MSR (Multi-step Retry)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev_cluster_deployment/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev_cluster_deployment/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dpdk/">Ceph messenger DPDKStack</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kclient/">Testing changes to the Linux Kernel CephFS driver</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kclient/#step-one-build-the-kernel">Step One: build the kernel</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kclient/#step-two-create-a-vm">Step Two: create a VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kclient/#step-three-networking-the-vm">Step Three: Networking the VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../libcephfs_proxy/">Design of the libcephfs proxy</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mempool_accounting/">What is a mempool?</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mempool_accounting/#some-common-mempools-that-we-can-track">Some common mempools that we can track</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">Crimson developer documentation</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">Crimson</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#building-crimson">Building Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#testing-crimson-with-cephadm">Testing crimson with cephadm</a></li>
<li class="toctree-l4"><a class="reference internal" href="#configure-crimson-with-bluestore">Configure Crimson with Bluestore</a></li>
<li class="toctree-l4"><a class="reference internal" href="#running-crimson">Running Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#enabling-crimson">Enabling Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#vstart-sh">vstart.sh</a></li>
<li class="toctree-l4"><a class="reference internal" href="#object-store-backend">Object Store Backend</a></li>
<li class="toctree-l4"><a class="reference internal" href="#profiling-crimson">Profiling Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#hacking-crimson">Hacking Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#debugging-crimson">Debugging Crimson</a></li>
<li class="toctree-l4"><a class="reference internal" href="#code-walkthroughs">Code Walkthroughs</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../osd/">OSDState</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pipeline/">The ClientRequest Pipeline</a></li>
<li class="toctree-l3"><a class="reference internal" href="../error-handling/">Error Handling</a></li>
<li class="toctree-l3"><a class="reference internal" href="../backfillmachine/">BackfillMachine</a></li>
<li class="toctree-l3"><a class="reference internal" href="../poseidonstore/">PoseidonStore</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="crimson">
<h1>crimson<a class="headerlink" href="#crimson" title="Permalink to this heading"></a></h1>
<p>Crimson is the code name of <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>, which is the next
generation <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>. It improves performance when using fast network
and storage devices, employing state-of-the-art technologies including
DPDK and SPDK. BlueStore continues to support HDDs and slower SSDs.
Crimson aims to be backward compatible with the classic <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>.</p>
<section id="building-crimson">
<h2>Building Crimson<a class="headerlink" href="#building-crimson" title="Permalink to this heading"></a></h2>
<p>Crimson is not enabled by default. Enable it at build time by running:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nv">WITH_SEASTAR</span><span class="o">=</span><span class="nb">true</span><span class="w"> </span>./install-deps.sh
<span class="gp">$ </span>./do_cmake.sh<span class="w"> </span>-DWITH_SEASTAR<span class="o">=</span>ON
</pre></div>
</div>
<p>Please note, <a class="reference external" href="https://github.com/google/sanitizers/wiki/AddressSanitizer">ASan</a> is enabled by default if Crimson is built from a source
cloned using <code class="docutils literal notranslate"><span class="pre">git</span></code>.</p>
</section>
<section id="testing-crimson-with-cephadm">
<h2>Testing crimson with cephadm<a class="headerlink" href="#testing-crimson-with-cephadm" title="Permalink to this heading"></a></h2>
<p>The Ceph CI/CD pipeline builds containers with
<code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> substituted for <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>.</p>
<p>Once a branch at commit &lt;sha1&gt; has been built and is available in
<code class="docutils literal notranslate"><span class="pre">shaman</span></code>, you can deploy it using the cephadm instructions outlined
in <a class="reference internal" href="../../../cephadm/#cephadm"><span class="std std-ref">Cephadm</span></a> with the following adaptations.</p>
<p>First, while performing the initial bootstrap, use the <code class="docutils literal notranslate"><span class="pre">--image</span></code> flag to
use a Crimson build:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "# ";
}
</style><span class="prompt1">cephadm<span class="w"> </span>--image<span class="w"> </span>quay.ceph.io/ceph-ci/ceph:&lt;sha1&gt;-crimson<span class="w"> </span>--allow-mismatched-release<span class="w"> </span>bootstrap<span class="w"> </span>...</span>
</pre></div></div><p>You’ll likely need to supply the <code class="docutils literal notranslate"><span class="pre">--allow-mismatched-release</span></code> flag to
use a non-release branch.</p>
</section>
<section id="configure-crimson-with-bluestore">
<h2>Configure Crimson with Bluestore<a class="headerlink" href="#configure-crimson-with-bluestore" title="Permalink to this heading"></a></h2>
<p>As Bluestore is not a Crimson native <a class="reference internal" href="#object-store-backend">object store backend</a>,
deploying Crimson with Bluestore as the back end requires setting
one of the two following configuration options:</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<ol class="arabic simple">
<li><p>These two options, along with <code class="docutils literal notranslate"><span class="pre">crimson_alien_op_num_threads</span></code>,
can’t be changed after deployment.</p></li>
<li><p><a class="reference internal" href="#vstart-sh">vstart.sh</a> sets these options using the <code class="docutils literal notranslate"><span class="pre">--crimson-smp</span></code> flag.</p></li>
</ol>
</div>
<ol class="arabic">
<li><p><code class="docutils literal notranslate"><span class="pre">crimson_seastar_num_threads</span></code></p>
<p>In order to allow easier cluster deployments, this option can be used
instead of setting the CPU mask manually for each OSD.</p>
<p>It’s recommended to let the <strong>number of OSDs on each host</strong> multiplied by
<code class="docutils literal notranslate"><span class="pre">crimson_seastar_num_threads</span></code> to be less than the node’s number of CPU
cores (<code class="docutils literal notranslate"><span class="pre">nproc</span></code>).</p>
<p>For example, for deploying two nodes with eight CPU cores and two OSDs each:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">conf</span><span class="p">:</span>
<span class="w">  </span><span class="c1"># Global to all OSDs</span>
<span class="w">  </span><span class="nt">osd</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar num threads</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<ol class="arabic simple">
<li><p>For optimal performance <code class="docutils literal notranslate"><span class="pre">crimson_seastar_cpu_cores</span></code> should be set instead.</p></li>
</ol>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">crimson_seastar_cpu_cores</span></code> and <code class="docutils literal notranslate"><span class="pre">crimson_alien_thread_cpu_cores</span></code>.</p>
<p>Explicitly set the CPU core allocation for each <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>
and for the BlueStore back end. It’s recommended for each set to be mutually exclusive.</p>
<p>For example, for deploying two nodes with eight CPU cores and two OSDs each:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">conf</span><span class="p">:</span>
<span class="w">  </span><span class="c1"># Both nodes</span>
<span class="w">  </span><span class="nt">osd</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson alien thread cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">6-7</span>

<span class="w">  </span><span class="c1"># First node</span>
<span class="w">  </span><span class="nt">osd.0</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0-2</span>
<span class="w">  </span><span class="nt">osd.1</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3-5</span>

<span class="w">  </span><span class="c1"># Second node</span>
<span class="w">  </span><span class="nt">osd.2</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0-2</span>
<span class="w">  </span><span class="nt">osd.3</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3-5</span>
</pre></div>
</div>
<p>For a single node with eight node and three OSDs:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">conf</span><span class="p">:</span>
<span class="w">  </span><span class="nt">osd</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson alien thread cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">6-7</span>
<span class="w">  </span><span class="nt">osd.0</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0-1</span>
<span class="w">  </span><span class="nt">osd.1</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2-3</span>
<span class="w">  </span><span class="nt">osd.2</span><span class="p">:</span>
<span class="w">    </span><span class="nt">crimson seastar cpu cores</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">4-5</span>
</pre></div>
</div>
</li>
</ol>
</section>
<section id="running-crimson">
<h2>Running Crimson<a class="headerlink" href="#running-crimson" title="Permalink to this heading"></a></h2>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Crimson is in a tech preview stage.
As you might expect, Crimson does not yet have as extensive a feature set as does ceph-osd.
Malfunctions including crashes and data loss are to be expected.</p>
</div>
</section>
<section id="enabling-crimson">
<h2>Enabling Crimson<a class="headerlink" href="#enabling-crimson" title="Permalink to this heading"></a></h2>
<p>After building Crimson and starting your cluster, but prior to deploying OSDs, you’ll need to
<a class="reference internal" href="#configure-crimson-with-bluestore">Configure Crimson with Bluestore</a> and enable Crimson to
direct the default pools to be created as Crimson pools.  You can proceed by running the following after you have a running cluster:</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><a class="reference internal" href="#vstart-sh">vstart.sh</a> enables crimson automatically when <cite>--crimson</cite> is used.</p>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>global<span class="w"> </span><span class="s1">&#39;enable_experimental_unrecoverable_data_corrupting_features&#39;</span><span class="w"> </span>crimson</span>
<span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>set-allow-crimson<span class="w"> </span>--yes-i-really-mean-it</span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mon<span class="w"> </span>osd_pool_default_crimson<span class="w"> </span><span class="nb">true</span></span>
</pre></div></div><p>The first command enables the <code class="docutils literal notranslate"><span class="pre">crimson</span></code> experimental feature.</p>
<p>The second enables the <code class="docutils literal notranslate"><span class="pre">allow_crimson</span></code> OSDMap flag.  The monitor will
not allow <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> to boot without that flag.</p>
<p>The last causes pools to be created by default with the <code class="docutils literal notranslate"><span class="pre">crimson</span></code> flag.
Crimson pools are restricted to operations supported by Crimson.
<code class="docutils literal notranslate"><span class="pre">Crimson-osd</span></code> won’t instantiate PGs from non-Crimson pools.</p>
</section>
<section id="vstart-sh">
<h2>vstart.sh<a class="headerlink" href="#vstart-sh" title="Permalink to this heading"></a></h2>
<p>The following options can be used with <code class="docutils literal notranslate"><span class="pre">vstart.sh</span></code>.</p>
<dl>
<dt><code class="docutils literal notranslate"><span class="pre">--crimson</span></code></dt><dd><p>Start <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> instead of <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--nodaemon</span></code></dt><dd><p>Do not daemonize the service.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--redirect-output</span></code></dt><dd><p>Redirect the <code class="docutils literal notranslate"><span class="pre">stdout</span></code> and <code class="docutils literal notranslate"><span class="pre">stderr</span></code> to <code class="docutils literal notranslate"><span class="pre">out/$type.$num.stdout</span></code>.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--osd-args</span></code></dt><dd><p>Pass extra command line options to <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> or <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>.
This is useful for passing Seastar options to <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>. For
example, one can supply <code class="docutils literal notranslate"><span class="pre">--osd-args</span> <span class="pre">&quot;--memory</span> <span class="pre">2G&quot;</span></code> to set the amount of
memory to use. Please refer to the output of:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">crimson-osd --help-seastar</span>
</pre></div>
</div>
<p>for additional Seastar-specific command line options.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--crimson-smp</span></code></dt><dd><p>The number of cores to use for each OSD.
If BlueStore is used, the balance of available cores
(as determined by <cite>nproc</cite>) will be assigned to the object store.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--bluestore</span></code></dt><dd><p>Use the alienized BlueStore as the object store backend. This is the default (see below section on the <a class="reference internal" href="#object-store-backend">object store backend</a> for more details)</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--cyanstore</span></code></dt><dd><p>Use CyanStore as the object store backend.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--memstore</span></code></dt><dd><p>Use the alienized MemStore as the object store backend.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--seastore</span></code></dt><dd><p>Use SeaStore as the back end object store.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--seastore-devs</span></code></dt><dd><p>Specify the block device used by SeaStore.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--seastore-secondary-devs</span></code></dt><dd><p>Optional.  SeaStore supports multiple devices.  Enable this feature by
passing the block device to this option.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--seastore-secondary-devs-type</span></code></dt><dd><p>Optional.  Specify the type of secondary devices.  When the secondary
device is slower than main device passed to <code class="docutils literal notranslate"><span class="pre">--seastore-devs</span></code>, the cold
data in faster device will be evicted to the slower devices over time.
Valid types include <code class="docutils literal notranslate"><span class="pre">HDD</span></code>, <code class="docutils literal notranslate"><span class="pre">SSD``(default),</span> <span class="pre">``ZNS</span></code>, and <code class="docutils literal notranslate"><span class="pre">RANDOM_BLOCK_SSD</span></code>
Note secondary devices should not be faster than the main device.</p>
</dd>
</dl>
<p>To start a cluster with a single Crimson node, run:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$  </span><span class="nv">MGR</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">MON</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">OSD</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">MDS</span><span class="o">=</span><span class="m">0</span><span class="w"> </span><span class="nv">RGW</span><span class="o">=</span><span class="m">0</span><span class="w"> </span>../src/vstart.sh<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--without-dashboard<span class="w"> </span>--bluestore<span class="w"> </span>--crimson<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--redirect-output
</pre></div>
</div>
<p>Another SeaStore example:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$  </span><span class="nv">MGR</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">MON</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">OSD</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="nv">MDS</span><span class="o">=</span><span class="m">0</span><span class="w"> </span><span class="nv">RGW</span><span class="o">=</span><span class="m">0</span><span class="w"> </span>../src/vstart.sh<span class="w"> </span>-n<span class="w"> </span>-x<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--without-dashboard<span class="w"> </span>--seastore<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--crimson<span class="w"> </span>--redirect-output<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--seastore-devs<span class="w"> </span>/dev/sda<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--seastore-secondary-devs<span class="w"> </span>/dev/sdb<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--seastore-secondary-devs-type<span class="w"> </span>HDD
</pre></div>
</div>
<p>Stop this <code class="docutils literal notranslate"><span class="pre">vstart</span></code> cluster by running:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>../src/stop.sh<span class="w"> </span>--crimson
</pre></div>
</div>
</section>
<section id="object-store-backend">
<h2>Object Store Backend<a class="headerlink" href="#object-store-backend" title="Permalink to this heading"></a></h2>
<p>At the moment, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> offers both native and alienized object store
backends. The native object store backends perform IO using the SeaStar reactor.
They are:</p>
<dl class="describe">
<dt class="sig sig-object">
<span class="sig-name descname"><span class="pre">cyanstore</span></span></dt>
<dd><p>CyanStore is modeled after memstore in the classic OSD.</p>
</dd></dl>

<dl class="describe">
<dt class="sig sig-object">
<span class="sig-name descname"><span class="pre">seastore</span></span></dt>
<dd><p>Seastore is still under active development.</p>
</dd></dl>

<p>The alienized object store backends are backed by a thread pool, which
is a proxy of the alienstore adaptor running in Seastar. The proxy issues
requests to object stores running in alien threads, i.e., worker threads not
managed by the Seastar framework. They are:</p>
<dl class="describe">
<dt class="sig sig-object">
<span class="sig-name descname"><span class="pre">memstore</span></span></dt>
<dd><p>The memory backend object store</p>
</dd></dl>

<dl class="describe">
<dt class="sig sig-object">
<span class="sig-name descname"><span class="pre">bluestore</span></span></dt>
<dd><p>The object store used by the classic <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code></p>
</dd></dl>

<section id="daemonize">
<h3>daemonize<a class="headerlink" href="#daemonize" title="Permalink to this heading"></a></h3>
<p>Unlike <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code> does not daemonize itself even if the
<code class="docutils literal notranslate"><span class="pre">daemonize</span></code> option is enabled. In order to read this option, <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>
needs to ready its config sharded service, but this sharded service lives
in the Seastar reactor. If we fork a child process and exit the parent after
starting the Seastar engine, that will leave us with a single thread which is
a replica of the thread that called <a class="reference external" href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html">fork()</a>. Tackling this problem in Crimson
would unnecessarily complicate the code.</p>
<p>Since supported GNU/Linux distributions use <code class="docutils literal notranslate"><span class="pre">systemd</span></code>, which is able to
daemonize processes, there is no need to daemonize ourselves.
Those using sysvinit can use <code class="docutils literal notranslate"><span class="pre">start-stop-daemon</span></code> to daemonize <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>.
If this is does not work out, a helper utility may be devised.</p>
</section>
<section id="logging">
<h3>logging<a class="headerlink" href="#logging" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">Crimson-osd</span></code> currently uses the logging utility offered by Seastar. See
<code class="docutils literal notranslate"><span class="pre">src/common/dout.h</span></code> for the mapping between Ceph logging levels to
the severity levels in Seastar. For instance, messages sent to <code class="docutils literal notranslate"><span class="pre">derr</span></code>
will be issued using <code class="docutils literal notranslate"><span class="pre">logger::error()</span></code>, and the messages with a debug level
greater than <code class="docutils literal notranslate"><span class="pre">20</span></code> will be issued using <code class="docutils literal notranslate"><span class="pre">logger::trace()</span></code>.</p>
<table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p>ceph</p></td>
<td><p>seastar</p></td>
</tr>
<tr class="row-even"><td><p>&lt; 0</p></td>
<td><p>error</p></td>
</tr>
<tr class="row-odd"><td><p>0</p></td>
<td><p>warn</p></td>
</tr>
<tr class="row-even"><td><p>[1, 6)</p></td>
<td><p>info</p></td>
</tr>
<tr class="row-odd"><td><p>[6, 20]</p></td>
<td><p>debug</p></td>
</tr>
<tr class="row-even"><td><p>&gt;  20</p></td>
<td><p>trace</p></td>
</tr>
</tbody>
</table>
<p>Note that <code class="docutils literal notranslate"><span class="pre">crimson-osd</span></code>
does not send log messages directly to a specified <code class="docutils literal notranslate"><span class="pre">log_file</span></code>. It writes
the logging messages to stdout and/or syslog. This behavior can be
changed using <code class="docutils literal notranslate"><span class="pre">--log-to-stdout</span></code> and <code class="docutils literal notranslate"><span class="pre">--log-to-syslog</span></code> command line
options. By default, <code class="docutils literal notranslate"><span class="pre">log-to-stdout</span></code> is enabled, and <code class="docutils literal notranslate"><span class="pre">--log-to-syslog</span></code> is disabled.
Metrics and Tracing
===================</p>
<p>Crimson offers three ways to report stats and metrics.</p>
</section>
<section id="pg-stats-reported-to-mgr">
<h3>PG stats reported to mgr<a class="headerlink" href="#pg-stats-reported-to-mgr" title="Permalink to this heading"></a></h3>
<p>Crimson collects the per-pg, per-pool, and per-osd stats in a <cite>MPGStats</cite>
message which is sent to the Ceph Managers. Manager modules can query
them using the <cite>MgrModule.get()</cite> method.</p>
</section>
<section id="asock-command">
<h3>Asock command<a class="headerlink" href="#asock-command" title="Permalink to this heading"></a></h3>
<p>An admin socket command is offered for dumping metrics:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ceph<span class="w"> </span>tell<span class="w"> </span>osd.0<span class="w"> </span>dump_metrics
<span class="gp">$ </span>ceph<span class="w"> </span>tell<span class="w"> </span>osd.0<span class="w"> </span>dump_metrics<span class="w"> </span>reactor_utilization
</pre></div>
</div>
<p>Here <cite>reactor_utilization</cite> is an optional string allowing us to filter
the dumped metrics by prefix.</p>
</section>
<section id="prometheus-text-protocol">
<h3>Prometheus text protocol<a class="headerlink" href="#prometheus-text-protocol" title="Permalink to this heading"></a></h3>
<p>The listening port and address can be configured using the command line options of
<cite>--prometheus_port</cite>
see <a class="reference external" href="https://github.com/scylladb/seastar/blob/master/doc/prometheus.md">Prometheus</a> for more details.</p>
</section>
</section>
<section id="profiling-crimson">
<h2>Profiling Crimson<a class="headerlink" href="#profiling-crimson" title="Permalink to this heading"></a></h2>
<section id="fio">
<h3>Fio<a class="headerlink" href="#fio" title="Permalink to this heading"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code> exposes configurable <code class="docutils literal notranslate"><span class="pre">FuturizedStore</span></code> internals as an
NBD server for use with <code class="docutils literal notranslate"><span class="pre">fio</span></code>.</p>
<p>In order to use <code class="docutils literal notranslate"><span class="pre">fio</span></code> to test <code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code>, perform the below steps.</p>
<ol class="arabic">
<li><p>You will need to install <code class="docutils literal notranslate"><span class="pre">libnbd</span></code>, and compile it into <code class="docutils literal notranslate"><span class="pre">fio</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt2:before {
  content: "$ ";
}
</style><span class="prompt2">apt-get<span class="w"> </span>install<span class="w"> </span>libnbd-dev</span>
<span class="prompt2">git<span class="w"> </span>clone<span class="w"> </span>git://git.kernel.dk/fio.git</span>
<span class="prompt2"><span class="nb">cd</span><span class="w"> </span>fio</span>
<span class="prompt2">./configure<span class="w"> </span>--enable-libnbd</span>
<span class="prompt2">make</span>
</pre></div></div></li>
<li><p>Build <code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2"><span class="nb">cd</span><span class="w"> </span>build</span>
<span class="prompt2">ninja<span class="w"> </span>crimson-store-nbd</span>
</pre></div></div></li>
<li><p>Run the <code class="docutils literal notranslate"><span class="pre">crimson-store-nbd</span></code> server with a block device. Specify
the path to the raw device, for example <code class="docutils literal notranslate"><span class="pre">/dev/nvme1n1</span></code>, in place of the created
file for testing with a block device.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2"><span class="nb">export</span><span class="w"> </span><span class="nv">disk_img</span><span class="o">=</span>/tmp/disk.img</span>
<span class="prompt2"><span class="nb">export</span><span class="w"> </span><span class="nv">unix_socket</span><span class="o">=</span>/tmp/store_nbd_socket.sock</span>
<span class="prompt2">rm<span class="w"> </span>-f<span class="w"> </span><span class="nv">$disk_img</span><span class="w"> </span><span class="nv">$unix_socket</span></span>
<span class="prompt2">truncate<span class="w"> </span>-s<span class="w"> </span>512M<span class="w"> </span><span class="nv">$disk_img</span></span>
<span class="prompt2">./bin/crimson-store-nbd<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--device-path<span class="w"> </span><span class="nv">$disk_img</span><span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--smp<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--mkfs<span class="w"> </span><span class="nb">true</span><span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--type<span class="w"> </span>transaction_manager<span class="w"> </span><span class="se">\</span>
<span class="w">  </span>--uds-path<span class="w"> </span><span class="si">${</span><span class="nv">unix_socket</span><span class="si">}</span><span class="w"> </span><span class="p">&amp;</span></span>
</pre></div></div><p>Below are descriptions of these command line arguments:</p>
<dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">--smp</span></code></dt><dd><p>The number of CPU cores to use (Symmetric MultiProcessor)</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--mkfs</span></code></dt><dd><p>Initialize the device first.</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">--type</span></code></dt><dd><p>The back end to use. If <code class="docutils literal notranslate"><span class="pre">transaction_manager</span></code> is specified, SeaStore’s
<code class="docutils literal notranslate"><span class="pre">TransactionManager</span></code> and <code class="docutils literal notranslate"><span class="pre">BlockSegmentManager</span></code> are used to emulate a
block device. Otherwise, this option is used to choose a backend of
<code class="docutils literal notranslate"><span class="pre">FuturizedStore</span></code>, where the whole “device” is divided into multiple
fixed-size objects whose size is specified by <code class="docutils literal notranslate"><span class="pre">--object-size</span></code>. So, if
you are only interested in testing the lower-level implementation of
SeaStore like logical address translation layer and garbage collection
without the object store semantics, <code class="docutils literal notranslate"><span class="pre">transaction_manager</span></code> would be a
better choice.</p>
</dd>
</dl>
</li>
<li><p>Create a <code class="docutils literal notranslate"><span class="pre">fio</span></code> job file named <code class="docutils literal notranslate"><span class="pre">nbd.fio</span></code></p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="k">[global]</span>
<span class="na">ioengine</span><span class="o">=</span><span class="s">nbd</span>
<span class="na">uri</span><span class="o">=</span><span class="s">nbd+unix:///?socket=${unix_socket}</span>
<span class="na">rw</span><span class="o">=</span><span class="s">randrw</span>
<span class="na">time_based</span>
<span class="na">runtime</span><span class="o">=</span><span class="s">120</span>
<span class="na">group_reporting</span>
<span class="na">iodepth</span><span class="o">=</span><span class="s">1</span>
<span class="na">size</span><span class="o">=</span><span class="s">512M</span>

<span class="k">[job0]</span>
<span class="na">offset</span><span class="o">=</span><span class="s">0</span>
</pre></div>
</div>
</li>
<li><p>Test the Crimson object store, using the custom <code class="docutils literal notranslate"><span class="pre">fio</span></code> built just now</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2">./fio<span class="w"> </span>nbd.fio</span>
</pre></div></div></li>
</ol>
</section>
<section id="cbt">
<h3>CBT<a class="headerlink" href="#cbt" title="Permalink to this heading"></a></h3>
<p>We can use <a class="reference external" href="https://github.com/ceph/cbt">cbt</a> for performance tests:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>git<span class="w"> </span>checkout<span class="w"> </span>main
<span class="gp">$ </span>make<span class="w"> </span>crimson-osd
<span class="gp">$ </span>../src/script/run-cbt.sh<span class="w"> </span>--cbt<span class="w"> </span>~/dev/cbt<span class="w"> </span>-a<span class="w"> </span>/tmp/baseline<span class="w"> </span>../src/test/crimson/cbt/radosbench_4K_read.yaml
<span class="gp">$ </span>git<span class="w"> </span>checkout<span class="w"> </span>yet-another-pr
<span class="gp">$ </span>make<span class="w"> </span>crimson-osd
<span class="gp">$ </span>../src/script/run-cbt.sh<span class="w"> </span>--cbt<span class="w"> </span>~/dev/cbt<span class="w"> </span>-a<span class="w"> </span>/tmp/yap<span class="w"> </span>../src/test/crimson/cbt/radosbench_4K_read.yaml
<span class="gp">$ </span>~/dev/cbt/compare.py<span class="w"> </span>-b<span class="w"> </span>/tmp/baseline<span class="w"> </span>-a<span class="w"> </span>/tmp/yap<span class="w"> </span>-v
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/0: bandwidth: (or (greater) (near 0.05)):: 0.183165/0.186155  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/0: iops_avg: (or (greater) (near 0.05)):: 46.0/47.0  =&gt; accepted</span>
<span class="go">19:48:23 - WARNING  - cbt      - prefill/gen8/0: iops_stddev: (or (less) (near 0.05)):: 10.4403/6.65833  =&gt; rejected</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.340868/0.333712  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: bandwidth: (or (greater) (near 0.05)):: 0.190447/0.177619  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_avg: (or (greater) (near 0.05)):: 48.0/45.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: iops_stddev: (or (less) (near 0.05)):: 6.1101/9.81495  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - prefill/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.325163/0.350251  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: bandwidth: (or (greater) (near 0.05)):: 1.24654/1.22336  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: iops_avg: (or (greater) (near 0.05)):: 319.0/313.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/0: latency_avg: (or (less) (near 0.05)):: 0.0497733/0.0509029  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: bandwidth: (or (greater) (near 0.05)):: 1.22717/1.11372  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: iops_avg: (or (greater) (near 0.05)):: 314.0/285.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: iops_stddev: (or (less) (near 0.05)):: 0.0/0.0  =&gt; accepted</span>
<span class="go">19:48:23 - INFO     - cbt      - seq/gen8/1: latency_avg: (or (less) (near 0.05)):: 0.0508262/0.0557337  =&gt; accepted</span>
<span class="go">19:48:23 - WARNING  - cbt      - 1 tests failed out of 16</span>
</pre></div>
</div>
<p>Here we compile and run the same test against two branches: <code class="docutils literal notranslate"><span class="pre">main</span></code> and <code class="docutils literal notranslate"><span class="pre">yet-another-pr</span></code>.
We then compare the results. Along with every test case, a set of rules is defined to check for
performance regressions when comparing the sets of test results. If a possible regression is found, the rule and
corresponding test results are highlighted.</p>
</section>
</section>
<section id="hacking-crimson">
<h2>Hacking Crimson<a class="headerlink" href="#hacking-crimson" title="Permalink to this heading"></a></h2>
<section id="seastar-documents">
<h3>Seastar Documents<a class="headerlink" href="#seastar-documents" title="Permalink to this heading"></a></h3>
<p>See <a class="reference external" href="https://github.com/scylladb/seastar/blob/master/doc/tutorial.md">Seastar Tutorial</a> .
Or build a browsable version and start an HTTP server:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>seastar
<span class="gp">$ </span>./configure.py<span class="w"> </span>--mode<span class="w"> </span>debug
<span class="gp">$ </span>ninja<span class="w"> </span>-C<span class="w"> </span>build/debug<span class="w"> </span>docs
<span class="gp">$ </span>python3<span class="w"> </span>-m<span class="w"> </span>http.server<span class="w"> </span>-d<span class="w"> </span>build/debug/doc/html
</pre></div>
</div>
<p>You might want to install <code class="docutils literal notranslate"><span class="pre">pandoc</span></code> and other dependencies beforehand.</p>
</section>
</section>
<section id="debugging-crimson">
<h2>Debugging Crimson<a class="headerlink" href="#debugging-crimson" title="Permalink to this heading"></a></h2>
<section id="debugging-with-gdb">
<h3>Debugging with GDB<a class="headerlink" href="#debugging-with-gdb" title="Permalink to this heading"></a></h3>
<p>The <a class="reference external" href="https://github.com/scylladb/scylla/blob/master/docs/dev/debugging.md#tips-and-tricks">tips</a> for debugging Scylla also apply to Crimson.</p>
</section>
<section id="human-readable-backtraces-with-addr2line">
<h3>Human-readable backtraces with addr2line<a class="headerlink" href="#human-readable-backtraces-with-addr2line" title="Permalink to this heading"></a></h3>
<p>When a Seastar application crashes, it leaves us with a backtrace of addresses, like:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">Segmentation fault.</span>
<span class="go">Backtrace:</span>
<span class="go">  0x00000000108254aa</span>
<span class="go">  0x00000000107f74b9</span>
<span class="go">  0x00000000105366cc</span>
<span class="go">  0x000000001053682c</span>
<span class="go">  0x00000000105d2c2e</span>
<span class="go">  0x0000000010629b96</span>
<span class="go">  0x0000000010629c31</span>
<span class="go">  0x00002a02ebd8272f</span>
<span class="go">  0x00000000105d93ee</span>
<span class="go">  0x00000000103eff59</span>
<span class="go">  0x000000000d9c1d0a</span>
<span class="go">  /lib/x86_64-linux-gnu/libc.so.6+0x000000000002409a</span>
<span class="go">  0x000000000d833ac9</span>
<span class="go">Segmentation fault</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">seastar-addr2line</span></code> utility provided by Seastar can be used to map these
addresses to functions. The script expects input on <code class="docutils literal notranslate"><span class="pre">stdin</span></code>,
so we need to copy and paste the above addresses, then send EOF by inputting
<code class="docutils literal notranslate"><span class="pre">control-D</span></code> in the terminal.  One might  use <code class="docutils literal notranslate"><span class="pre">echo</span></code> or <code class="docutils literal notranslate"><span class="pre">cat</span></code> instead:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>../src/seastar/scripts/seastar-addr2line<span class="w"> </span>-e<span class="w"> </span>bin/crimson-osd

<span class="go">  0x00000000108254aa</span>
<span class="go">  0x00000000107f74b9</span>
<span class="go">  0x00000000105366cc</span>
<span class="go">  0x000000001053682c</span>
<span class="go">  0x00000000105d2c2e</span>
<span class="go">  0x0000000010629b96</span>
<span class="go">  0x0000000010629c31</span>
<span class="go">  0x00002a02ebd8272f</span>
<span class="go">  0x00000000105d93ee</span>
<span class="go">  0x00000000103eff59</span>
<span class="go">  0x000000000d9c1d0a</span>
<span class="go">  0x00000000108254aa</span>
<span class="go">[Backtrace #0]</span>
<span class="go">seastar::backtrace_buffer::append_backtrace() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1136</span>
<span class="go">seastar::print_with_backtrace(seastar::backtrace_buffer&amp;) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1157</span>
<span class="go">seastar::print_with_backtrace(char const*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:1164</span>
<span class="go">seastar::sigsegv_action() at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5119</span>
<span class="go">seastar::install_oneshot_signal_handler&lt;11, &amp;seastar::sigsegv_action&gt;()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5105</span>
<span class="go">seastar::install_oneshot_signal_handler&lt;11, &amp;seastar::sigsegv_action&gt;()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5101</span>
<span class="go">?? ??:0</span>
<span class="go">seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config) at /home/kefu/dev/ceph/build/../src/seastar/src/core/reactor.cc:5418</span>
<span class="go">seastar::app_template::run_deprecated(int, char**, std::function&lt;void ()&gt;&amp;&amp;) at /home/kefu/dev/ceph/build/../src/seastar/src/core/app-template.cc:173 (discriminator 5)</span>
<span class="go">main at /home/kefu/dev/ceph/build/../src/crimson/osd/main.cc:131 (discriminator 1)</span>
</pre></div>
</div>
<p>Note that <code class="docutils literal notranslate"><span class="pre">seastar-addr2line</span></code> is able to extract addresses from
its input, so you can also paste the log messages as below:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">2020-07-22T11:37:04.500 INFO:teuthology.orchestra.run.smithi061.stderr:Backtrace:</span>
<span class="go">2020-07-22T11:37:04.500 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e78dbc</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e3e7f0</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e3e8b8</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  0x0000000000e3e985</span>
<span class="go">2020-07-22T11:37:04.501 INFO:teuthology.orchestra.run.smithi061.stderr:  /lib64/libpthread.so.0+0x0000000000012dbf</span>
</pre></div>
</div>
<p>Unlike the classic <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code>, Crimson does not print a human-readable backtrace when it
handles fatal signals like <cite>SIGSEGV</cite> or <cite>SIGABRT</cite>. It is also more complicated
with a stripped binary. So instead of planting a signal handler for
those signals into Crimson, we can use <cite>script/ceph-debug-docker.sh</cite> to map
addresses in the backtrace:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp"># </span>assuming<span class="w"> </span>you<span class="w"> </span>are<span class="w"> </span>under<span class="w"> </span>the<span class="w"> </span><span class="nb">source</span><span class="w"> </span>tree<span class="w"> </span>of<span class="w"> </span>ceph
<span class="gp">$ </span>./src/script/ceph-debug-docker.sh<span class="w">  </span>--flavor<span class="w"> </span>crimson<span class="w"> </span>master:27e237c137c330ebb82627166927b7681b20d0aa<span class="w"> </span>centos:8
<span class="go">....</span>
<span class="gp">[root@3deb50a8ad51 ~]# </span>wget<span class="w"> </span>-q<span class="w"> </span>https://raw.githubusercontent.com/scylladb/seastar/master/scripts/seastar-addr2line
<span class="gp">[root@3deb50a8ad51 ~]# </span>dnf<span class="w"> </span>install<span class="w"> </span>-q<span class="w"> </span>-y<span class="w"> </span>file
<span class="gp">[root@3deb50a8ad51 ~]# </span>python3<span class="w"> </span>seastar-addr2line<span class="w"> </span>-e<span class="w"> </span>/usr/bin/crimson-osd
<span class="gp"># </span>paste<span class="w"> </span>the<span class="w"> </span>backtrace<span class="w"> </span>here
</pre></div>
</div>
</section>
</section>
<section id="code-walkthroughs">
<h2>Code Walkthroughs<a class="headerlink" href="#code-walkthroughs" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=rtkrHk6grsg">Ceph Code Walkthroughs: Crimson</a></p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=0rr5oWDE2Ck">Ceph Code Walkthroughs: SeaStore</a></p></li>
</ul>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../" class="btn btn-neutral float-left" title="Crimson developer documentation" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../osd/" class="btn btn-neutral float-right" title="osd" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>